Page MenuHomePhabricator

[L] Update instrumentation & schema to track things for which we're currently relying on searchSatisfaction.js
Closed, ResolvedPublic

Description

The current 2.0.0 schema captures a session id from searchSatisfaction.js. That one, however, is not available on mobile.

Based on https://docs.google.com/spreadsheets/d/1drMnPt8mJa8rQRNCCpgkOo_0eQmdDlHDQnLIbjg8syU/edit#gid=0 & other discussions, we intended to use this session id for:

  1. cross-referencing with the searchsatisfaction schema to infer whether the user behind this session is registered or anon
  2. cross-referencing with the searchsatisfaction schema to infer whether the user behind this session is a bot
  3. counting the number of sessions with searchpreviews
  4. comparing the numbers of sessions with searchprevies vs. those where it isn't used

Since we will not be able to cross-reference with the searchsatisfaction schema, we can instead capture that information ourselves:

  1. add a new field is_anon to the searchpreviews schema where the return value of mw.user.isAnon() gets captured
  2. I think we can ignore bot - this would be annoying to implement (not available from the client side IIRC), and this is the kind of feature that a can't imagine a bot interacting with
  3. implement logic similar to the one in searchSatisfaction.js to generate new session ids; it will not be possible to cross-reference with the searchsatisfaction schema, but we can get to the same metrics if we use the same implementation. Note: also remove the documentation that says this value matches searchsatisfaction's event.searchSessionId
  4. add a new type (new-session) to the action enum in the searchpreviews schema to track the total number of sessions, so that we're able to cross-reference them with the amount of sessions the we track other actions for (or no actions)

1-3 are pretty straightforward.
For 4, we're going to need to add some more instrumentation. This new_session action should be called in the following circumstances:

  • prior to emitting any other event when we detect that the session_id has changed since the last event was logged (i.e. 10minute session time has passed)
  • when SearchVue is loaded on Special:Search & with results that support SearchVue when there is no active session yet

This essentially captures all session id where a user actively engages with search, even if they don't interact with SearchVue. Diffing session_ids for action=new_session against session_ids for all other actions will satisfy #4.

@cchen - can you confirm that all of the above works for you?

  • As part of QA for this task, ensure that all events are being properly tracked in mobile and re-check for desktop.

Event Timeline

@matthiasmullie i confirm the description works, and update the instrumentation document

CBogen renamed this task from Update instrumentation & schema to track things for which we're currently relying on searchSatisfaction.js to [L] Update instrumentation & schema to track things for which we're currently relying on searchSatisfaction.js.Jan 11 2023, 5:26 PM

Change 880953 had a related patch set uploaded (by Simone Cuomo; author: Simone Cuomo):

[schemas/event/secondary@master] Add new action to be able to track sessions

https://gerrit.wikimedia.org/r/880953

Change 881394 had a related patch set uploaded (by Simone Cuomo; author: Simone Cuomo):

[mediawiki/extensions/SearchVue@master] Replace searchSatisfaction in SearchPreview event implementation

https://gerrit.wikimedia.org/r/881394

The following have been implemented, please check the implementation details:

  • New event with action 'new-session' to be triggered on session load
  • Remove the dependencies to Session Satisfaction
  • Generate a new sessionId every 10 minutes of inactivity (if the user has the screen on but does not interact for 30 minutes 3 "new-session" event are sent
  • When sending a 'new-session' event, I am assigning a selectedIndex equal to "-1"

I am not sure my 3rd point is correct, as the AC did not specify to emit a new-session event, but I guess we need to have the new session if we have a new sessionId.

cc: @cchen @matthiasmullie

Change 880953 merged by jenkins-bot:

[schemas/event/secondary@master] Update searchPreview schema to be inline with required changes

https://gerrit.wikimedia.org/r/880953

  • Generate a new sessionId every 10 minutes of inactivity (if the user has the screen on but does not interact for 30 minutes 3 "new-session" event are sent

I am not sure my 3rd point is correct, as the AC did not specify to emit a new-session event, but I guess we need to have the new session if we have a new sessionId.

Sessions in searchSatisfaction.js work a little differently.

In searchSatisFaction.js, sessions persist across searches, as long as they happen within 10 minutes of the last search. If a user moves off of the search pages and comes back 6 minutes later to do another search, it's still part of the same search session (and extends the session with another 10 minutes)
It's not until there have been 10 minutes of no searches, that a new search starts a new session.

I've updated the patch to change that.
A 'new-session' event will be emitted for every new search session (even if no searchvue interaction will follow), but a new session will not be initiated until the user does a new search that is at least 10 minutes after their previous search. This matches searchSatisfaction's logic.
For every searchvue action logged, there should be 1 'new-session' action logged with a matching session id.
There will also be 'new-session' actions with session ids for which there are no other searchvue actions logged (when users didn't interact with searchvue during that session)

Change 881394 merged by jenkins-bot:

[mediawiki/extensions/SearchVue@master] Replace searchSatisfaction in SearchPreview event implementation

https://gerrit.wikimedia.org/r/881394

Etonkovidova subscribed.

Tracked searchsatisfaction schema on betalabs stream for eventlogging_SearchSatisfaction
According to the specs, the following is done and checked:

What was also checked

  • validation errors ( on @deployment-eventlog08:/srv/log/eventlogging/systemd$ tail -f eventlogging-processor@client-side-01.log) - no validation errors
  • this AC specifically mentions tracking mobile events - during my testing on betalabs the events from mobile were not displayed in betalabs stream for eventlogging_SearchSatisfaction

As part of QA for this task, ensure that all events are being properly tracked in mobile and re-check for desktop.

Btw, searchpreview schema mobile events are recorded now, e.g.:

{
"$schema": "/analytics/mediawiki/searchpreview/3.0.0",
"action": "open-searchpreview",
"result_display_position": 2,
"wiki_id": "enwiki",
"platform": "mobile",
"is_anon": false,
"session_id": "1f3c2a3c79dd00edac0bldkq5o2d",
"meta":
{
"stream": "mediawiki.searchpreview",
"domain": "en.wikipedia.beta.wmflabs.org",
"id": "860f80d2-4f39-42e6-9236-b7d255dc7da4",
"dt": "2023-01-31T21:02:08.292Z",
"request_id": "85fe4820-a1aa-11ed-be74-832ea79fb2f0",
"topic": "eqiad.mediawiki.searchpreview",
"partition": 0,
"offset": 3944

@cchen - I'm moving the task to Verify on Production. Checking in betalabs has some limitations, so the schema needs to be checked in production.

Thank you @Etonkovidova, I update the QA in production here T324707#8576312