Page MenuHomePhabricator
Create Task
Maniphest T164856

A/B Test: explore similar - verify data coming in is good
Closed, ResolvedPublic
Actions

Description

Once the test starts - let's be sure that the data gathered in event logging is what we want to see.

Details

Related Gerrit Patches:
mediawiki/extensions/CirrusSearch : masterAdding full URLs to Explore Similar API calls
mediawiki/extensions/CirrusSearch : wmf/1.30.0-wmf.9Adding full URLs to Explore Similar API calls

Related Objects
Search...

Event Timeline

debt created this task.May 9 2017, 5:34 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 9 2017, 5:34 PM
• chelsyx added a comment.Jul 7 2017, 12:32 AM

I checked the data at 4pm PDT July 6 and created the following summary tables. The main issue is that we don't have any 'esclick' event. This cause of this problem hasn't been found yet. :(

Test summary:

DaysEventsSessionsPage IDsSERP eventsUnique search queriesSERPsSame-wiki clicksOther clicks (sister search etc.)
7101311371445935003554285199741

Number of events:

groupeventn
explore_similar_controlcheckin2027
explore_similar_controlclick461
explore_similar_controliwclick4
explore_similar_controlsearchResultPage1828
explore_similar_controlssclick20
explore_similar_controlvisitPage444
explore_similar_testcheckin2253
explore_similar_testclick536
explore_similar_testesclick1 (created in debugging process)
explore_similar_testhover-off160
explore_similar_testhover-on194
explore_similar_testiwclick5
explore_similar_testsearchResultPage1672
explore_similar_testssclick11
explore_similar_testvisitPage515

Number of searches:

Test groupSearch sessionsSearches recorded
explore_similar_control7211,476
explore_similar_test6501,375
Total1,3712,851

Hover-on and hover-off:

eventsectionNo. resultsn
hover-offNANA160
hover-oncategories15
hover-oncategories28
hover-oncategories34
hover-oncategories410
hover-oncategories55
hover-oncategories61
hover-oncategories71
hover-oncategories82
hover-oncategories93
hover-oncategories1010
hover-onlanguages01
hover-onlanguages22
hover-onlanguages32
hover-onlanguages82
hover-onlanguages121
hover-onrelated02
hover-onrelated3135
Jdrewniak added a comment.Jul 10 2017, 8:46 PM

the no esclick events is certainly strange. I've checked the kafka logs, and I can see esclick events at least making it into there. Maybe they're not valid? could be an invalid param or something. I'm attaching a video and event output. We should check the logstash error logs to see if any errors are being generated.

https://www.dropbox.com/s/5zlrup1ob4l2763/hoveron-off-esclick.mp4?dl=0

jdrewniak@stat1002:~$ kafkacat -b kafka1012:9092 -C -t eventlogging_TestSearchSatisfaction2 2>/dev/null | grep "esclick\|hover-on\|hover-off"
{"event": {"action": "hover-on", "didYouMeanVisible": "no", "extraParams": "{\"hoverId\":\"e7f839n4fo9j4yljfr5\",\"section\":\"related\",\"results\":3}", "hitsReturned": 1306, "msToDisplayResults": 2533, "mwSessionId": "0dd7ad335387f888j4j2iekh", "pageViewId": "38da0f3084d892caj4yljbgq", "query": "~poops", "scroll": false, "searchSessionId": "explore_similar_test", "searchToken": "9w0c5l7hopad8rf0dcj1u2mmg", "source": "fulltext", "subTest": "explore_similar_test", "uniqueId": "5f763fc9a2a65008j4yljg2o"}, "recvFrom": "cp1066.eqiad.wmnet", "revision": 16909631, "schema": "TestSearchSatisfaction2", "seqId": 41899603, "timestamp": 1499718276, "userAgent": "{\"os_minor\": \"12\", \"is_bot\": false, \"os_major\": \"10\", \"device_family\": \"Other\", \"os_family\": \"Mac OS X\", \"browser_minor\": \"0\", \"wmf_app_version\": \"-\", \"browser_major\": \"59\", \"browser_family\": \"Chrome\", \"is_mediawiki\": false}", "uuid": "dfdc9b7eb62a5f44851464486a966cfb", "webHost": "en.wikipedia.org", "wiki": "enwiki"}
{"event": {"action": "hover-off", "didYouMeanVisible": "no", "extraParams": "{\"hoverId\":\"e7f839n4fo9j4yljfr5\"}", "hitsReturned": 1306, "msToDisplayResults": 2533, "mwSessionId": "0dd7ad335387f888j4j2iekh", "pageViewId": "38da0f3084d892caj4yljbgq", "query": "~poops", "scroll": false, "searchSessionId": "explore_similar_test", "searchToken": "9w0c5l7hopad8rf0dcj1u2mmg", "source": "fulltext", "subTest": "explore_similar_test", "uniqueId": "5beb4189c0880952j4yljhcc"}, "recvFrom": "cp1066.eqiad.wmnet", "revision": 16909631, "schema": "TestSearchSatisfaction2", "seqId": 41899614, "timestamp": 1499718277, "userAgent": "{\"os_minor\": \"12\", \"is_bot\": false, \"os_major\": \"10\", \"device_family\": \"Other\", \"os_family\": \"Mac OS X\", \"browser_minor\": \"0\", \"wmf_app_version\": \"-\", \"browser_major\": \"59\", \"browser_family\": \"Chrome\", \"is_mediawiki\": false}", "uuid": "76ed801fae9858309da7cb725b097bc5", "webHost": "en.wikipedia.org", "wiki": "enwiki"}
{"event": {"action": "hover-on", "didYouMeanVisible": "no", "extraParams": "{\"hoverId\":\"183wwez3kjrj4yljk88\",\"section\":\"related\",\"results\":3}", "hitsReturned": 1306, "msToDisplayResults": 2533, "mwSessionId": "0dd7ad335387f888j4j2iekh", "pageViewId": "38da0f3084d892caj4yljbgq", "query": "~poops", "scroll": false, "searchSessionId": "explore_similar_test", "searchToken": "9w0c5l7hopad8rf0dcj1u2mmg", "source": "fulltext", "subTest": "explore_similar_test", "uniqueId": "4957907bf8060609j4yljkho"}, "recvFrom": "cp1066.eqiad.wmnet", "revision": 16909631, "schema": "TestSearchSatisfaction2", "seqId": 41899639, "timestamp": 1499718282, "userAgent": "{\"os_minor\": \"12\", \"is_bot\": false, \"os_major\": \"10\", \"device_family\": \"Other\", \"os_family\": \"Mac OS X\", \"browser_minor\": \"0\", \"wmf_app_version\": \"-\", \"browser_major\": \"59\", \"browser_family\": \"Chrome\", \"is_mediawiki\": false}", "uuid": "6746ac9c644c5221b1af7e2712d8bf2e", "webHost": "en.wikipedia.org", "wiki": "enwiki"}
{"event": {"action": "esclick", "didYouMeanVisible": "no", "extraParams": "{\"hoverId\":\"183wwez3kjrj4yljk88\",\"section\":\"related\",\"result\":0}", "hitsReturned": 1306, "msToDisplayResults": 2533, "mwSessionId": "0dd7ad335387f888j4j2iekh", "pageViewId": "38da0f3084d892caj4yljbgq", "query": "~poops", "scroll": false, "searchSessionId": "explore_similar_test", "searchToken": "9w0c5l7hopad8rf0dcj1u2mmg", "source": "fulltext", "subTest": "explore_similar_test", "uniqueId": "92782fde07f8d86bj4yljqsv"}, "recvFrom": "cp1066.eqiad.wmnet", "revision": 16909631, "schema": "TestSearchSatisfaction2", "seqId": 41899674, "timestamp": 1499718290, "userAgent": "{\"os_minor\": \"12\", \"is_bot\": false, \"os_major\": \"10\", \"device_family\": \"Other\", \"os_family\": \"Mac OS X\", \"browser_minor\": \"0\", \"wmf_app_version\": \"-\", \"browser_major\": \"59\", \"browser_family\": \"Chrome\", \"is_mediawiki\": false}", "uuid": "3f260d6451ae580c8d9b2782b452efac", "webHost": "en.wikipedia.org", "wiki": "enwiki"}
gerritbot added a subscriber: gerritbot.Jul 11 2017, 4:17 PM

Change 364465 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[mediawiki/extensions/CirrusSearch@master] Adding full URLs to Explore Similar API calls

https://gerrit.wikimedia.org/r/364465

gerritbot added a comment.Jul 12 2017, 6:34 PM

Change 364465 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Adding full URLs to Explore Similar API calls

https://gerrit.wikimedia.org/r/364465

gerritbot added a comment.Jul 12 2017, 7:49 PM

Change 364834 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[mediawiki/extensions/CirrusSearch@wmf/1.30.0-wmf.9] Adding full URLs to Explore Similar API calls

https://gerrit.wikimedia.org/r/364834

gerritbot added a comment.Jul 12 2017, 11:20 PM

Change 364834 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@wmf/1.30.0-wmf.9] Adding full URLs to Explore Similar API calls

https://gerrit.wikimedia.org/r/364834

Stashbot mentioned this in T149809: [A/B Test] Add 'explore similar' pages, categories and suggested languages in search results.Jul 12 2017, 11:28 PM
Stashbot added a subscriber: Stashbot.

Mentioned in SAL (#wikimedia-operations) [2017-07-12T23:28:44Z] <thcipriani@tin> Synchronized php-1.30.0-wmf.9/extensions/CirrusSearch/resources/ext.cirrus.explore-similar.js: SWAT: [[gerrit:364834|Adding full URLs to Explore Similar API calls]] T149809 T164856 (duration: 00m 47s)

debt closed this task as Resolved.Jul 21 2017, 8:01 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL