Once the test starts - let's be sure that the data gathered in event logging is what we want to see.
Description
Description
| Status | Assigned | Task | ||
|---|---|---|---|---|
| Resolved | Jdrewniak | T149809 [A/B Test] Add 'explore similar' pages, categories and suggested languages in search results | ||
| Resolved | chelsyx | T164856 A/B Test: explore similar - verify data coming in is good |
Event Timeline
Comment Actions
I checked the data at 4pm PDT July 6 and created the following summary tables. The main issue is that we don't have any 'esclick' event. This cause of this problem hasn't been found yet. :(
Test summary:
| Days | Events | Sessions | Page IDs | SERP events | Unique search queries | SERPs | Same-wiki clicks | Other clicks (sister search etc.) |
| 7 | 10131 | 1371 | 4459 | 3500 | 3554 | 2851 | 997 | 41 |
Number of events:
| group | event | n |
| explore_similar_control | checkin | 2027 |
| explore_similar_control | click | 461 |
| explore_similar_control | iwclick | 4 |
| explore_similar_control | searchResultPage | 1828 |
| explore_similar_control | ssclick | 20 |
| explore_similar_control | visitPage | 444 |
| explore_similar_test | checkin | 2253 |
| explore_similar_test | click | 536 |
| explore_similar_test | esclick | 1 (created in debugging process) |
| explore_similar_test | hover-off | 160 |
| explore_similar_test | hover-on | 194 |
| explore_similar_test | iwclick | 5 |
| explore_similar_test | searchResultPage | 1672 |
| explore_similar_test | ssclick | 11 |
| explore_similar_test | visitPage | 515 |
Number of searches:
| Test group | Search sessions | Searches recorded |
| explore_similar_control | 721 | 1,476 |
| explore_similar_test | 650 | 1,375 |
| Total | 1,371 | 2,851 |
Hover-on and hover-off:
| event | section | No. results | n |
| hover-off | NA | NA | 160 |
| hover-on | categories | 1 | 5 |
| hover-on | categories | 2 | 8 |
| hover-on | categories | 3 | 4 |
| hover-on | categories | 4 | 10 |
| hover-on | categories | 5 | 5 |
| hover-on | categories | 6 | 1 |
| hover-on | categories | 7 | 1 |
| hover-on | categories | 8 | 2 |
| hover-on | categories | 9 | 3 |
| hover-on | categories | 10 | 10 |
| hover-on | languages | 0 | 1 |
| hover-on | languages | 2 | 2 |
| hover-on | languages | 3 | 2 |
| hover-on | languages | 8 | 2 |
| hover-on | languages | 12 | 1 |
| hover-on | related | 0 | 2 |
| hover-on | related | 3 | 135 |
Comment Actions
the no esclick events is certainly strange. I've checked the kafka logs, and I can see esclick events at least making it into there. Maybe they're not valid? could be an invalid param or something. I'm attaching a video and event output. We should check the logstash error logs to see if any errors are being generated.
https://www.dropbox.com/s/5zlrup1ob4l2763/hoveron-off-esclick.mp4?dl=0
jdrewniak@stat1002:~$ kafkacat -b kafka1012:9092 -C -t eventlogging_TestSearchSatisfaction2 2>/dev/null | grep "esclick\|hover-on\|hover-off"
{"event": {"action": "hover-on", "didYouMeanVisible": "no", "extraParams": "{\"hoverId\":\"e7f839n4fo9j4yljfr5\",\"section\":\"related\",\"results\":3}", "hitsReturned": 1306, "msToDisplayResults": 2533, "mwSessionId": "0dd7ad335387f888j4j2iekh", "pageViewId": "38da0f3084d892caj4yljbgq", "query": "~poops", "scroll": false, "searchSessionId": "explore_similar_test", "searchToken": "9w0c5l7hopad8rf0dcj1u2mmg", "source": "fulltext", "subTest": "explore_similar_test", "uniqueId": "5f763fc9a2a65008j4yljg2o"}, "recvFrom": "cp1066.eqiad.wmnet", "revision": 16909631, "schema": "TestSearchSatisfaction2", "seqId": 41899603, "timestamp": 1499718276, "userAgent": "{\"os_minor\": \"12\", \"is_bot\": false, \"os_major\": \"10\", \"device_family\": \"Other\", \"os_family\": \"Mac OS X\", \"browser_minor\": \"0\", \"wmf_app_version\": \"-\", \"browser_major\": \"59\", \"browser_family\": \"Chrome\", \"is_mediawiki\": false}", "uuid": "dfdc9b7eb62a5f44851464486a966cfb", "webHost": "en.wikipedia.org", "wiki": "enwiki"}
{"event": {"action": "hover-off", "didYouMeanVisible": "no", "extraParams": "{\"hoverId\":\"e7f839n4fo9j4yljfr5\"}", "hitsReturned": 1306, "msToDisplayResults": 2533, "mwSessionId": "0dd7ad335387f888j4j2iekh", "pageViewId": "38da0f3084d892caj4yljbgq", "query": "~poops", "scroll": false, "searchSessionId": "explore_similar_test", "searchToken": "9w0c5l7hopad8rf0dcj1u2mmg", "source": "fulltext", "subTest": "explore_similar_test", "uniqueId": "5beb4189c0880952j4yljhcc"}, "recvFrom": "cp1066.eqiad.wmnet", "revision": 16909631, "schema": "TestSearchSatisfaction2", "seqId": 41899614, "timestamp": 1499718277, "userAgent": "{\"os_minor\": \"12\", \"is_bot\": false, \"os_major\": \"10\", \"device_family\": \"Other\", \"os_family\": \"Mac OS X\", \"browser_minor\": \"0\", \"wmf_app_version\": \"-\", \"browser_major\": \"59\", \"browser_family\": \"Chrome\", \"is_mediawiki\": false}", "uuid": "76ed801fae9858309da7cb725b097bc5", "webHost": "en.wikipedia.org", "wiki": "enwiki"}
{"event": {"action": "hover-on", "didYouMeanVisible": "no", "extraParams": "{\"hoverId\":\"183wwez3kjrj4yljk88\",\"section\":\"related\",\"results\":3}", "hitsReturned": 1306, "msToDisplayResults": 2533, "mwSessionId": "0dd7ad335387f888j4j2iekh", "pageViewId": "38da0f3084d892caj4yljbgq", "query": "~poops", "scroll": false, "searchSessionId": "explore_similar_test", "searchToken": "9w0c5l7hopad8rf0dcj1u2mmg", "source": "fulltext", "subTest": "explore_similar_test", "uniqueId": "4957907bf8060609j4yljkho"}, "recvFrom": "cp1066.eqiad.wmnet", "revision": 16909631, "schema": "TestSearchSatisfaction2", "seqId": 41899639, "timestamp": 1499718282, "userAgent": "{\"os_minor\": \"12\", \"is_bot\": false, \"os_major\": \"10\", \"device_family\": \"Other\", \"os_family\": \"Mac OS X\", \"browser_minor\": \"0\", \"wmf_app_version\": \"-\", \"browser_major\": \"59\", \"browser_family\": \"Chrome\", \"is_mediawiki\": false}", "uuid": "6746ac9c644c5221b1af7e2712d8bf2e", "webHost": "en.wikipedia.org", "wiki": "enwiki"}
{"event": {"action": "esclick", "didYouMeanVisible": "no", "extraParams": "{\"hoverId\":\"183wwez3kjrj4yljk88\",\"section\":\"related\",\"result\":0}", "hitsReturned": 1306, "msToDisplayResults": 2533, "mwSessionId": "0dd7ad335387f888j4j2iekh", "pageViewId": "38da0f3084d892caj4yljbgq", "query": "~poops", "scroll": false, "searchSessionId": "explore_similar_test", "searchToken": "9w0c5l7hopad8rf0dcj1u2mmg", "source": "fulltext", "subTest": "explore_similar_test", "uniqueId": "92782fde07f8d86bj4yljqsv"}, "recvFrom": "cp1066.eqiad.wmnet", "revision": 16909631, "schema": "TestSearchSatisfaction2", "seqId": 41899674, "timestamp": 1499718290, "userAgent": "{\"os_minor\": \"12\", \"is_bot\": false, \"os_major\": \"10\", \"device_family\": \"Other\", \"os_family\": \"Mac OS X\", \"browser_minor\": \"0\", \"wmf_app_version\": \"-\", \"browser_major\": \"59\", \"browser_family\": \"Chrome\", \"is_mediawiki\": false}", "uuid": "3f260d6451ae580c8d9b2782b452efac", "webHost": "en.wikipedia.org", "wiki": "enwiki"}Comment Actions
Change 364465 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[mediawiki/extensions/CirrusSearch@master] Adding full URLs to Explore Similar API calls
Comment Actions
Change 364465 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Adding full URLs to Explore Similar API calls
Comment Actions
Change 364834 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[mediawiki/extensions/CirrusSearch@wmf/1.30.0-wmf.9] Adding full URLs to Explore Similar API calls
Comment Actions
Change 364834 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@wmf/1.30.0-wmf.9] Adding full URLs to Explore Similar API calls
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2017-07-12T23:28:44Z] <thcipriani@tin> Synchronized php-1.30.0-wmf.9/extensions/CirrusSearch/resources/ext.cirrus.explore-similar.js: SWAT: [[gerrit:364834|Adding full URLs to Explore Similar API calls]] T149809 T164856 (duration: 00m 47s)