Page MenuHomePhabricator

[SPIKE] Investigate what init type is associated with the sessions where a ready event is being emitted, but an anonymous user token is not being logged
Closed, ResolvedPublicSpike

Description

In an attempt to identify the underlying causes for the bucketing imbalances identified in T291308, we will us this ticket to investigate what init type is associated with the sessions where a ready event is being emitted, but an anonymous user token is not being logged.

Event Timeline

MNeisler moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.

@DLynch

Here's are the session init_types logged for ready events in the AB test where the anonymous user token was not assigned. These instances occur for both init_types but the majority in both test groups occur where the session init_type = page. Also, as I mentioned in my comment on T291308#7693635, the anonymous_user_token is only missing for bucketed ready events where event.integration = 'page' and editor_interface = 'wikitext'.

bucketintegrationanonymous_user_tokeneditor_interfaceinit_typenum_ready_events
controlpageNULLwikitextpage1255
controlpageNULLwikitextsection108
testpageNULLwikitextpage1228
testpageNULLwikitextsection81

Let me know if any other data would be helpful. Here's the query I used to collect the above data for reference:

WITH init_events AS (

SELECT
  event.editing_session_id as edit_attempt_id,
  event.init_type as init_type,
  wiki As init_wiki
FROM
  event.editattemptstep
WHERE
-- following deployment of patch
    YEAR = 2022
    AND month = 02
    AND day >= 03
-- inits events on desktop
    AND event.platform = 'desktop'
    AND event.action = 'init'
-- only talk pages
    AND event.page_ns % 2 = 1
--by anon not bot users
    AND useragent.is_bot = false
    AND event.user_id = 0 
   AND event.user_class = 'IP'
--- test wikis
    AND wiki IN ('amwiki', 'bnwiki', 'zhwiki', 'nlwiki', 'arzwiki', 'frwiki', 'hewiki', 'hiwiki',
    'idwiki', 'itwiki', 'jawiki', 'kowiki', 'omwiki', 'fawiki', 'plwiki', 'ptwiki', 'eswiki', 'thwiki',
    'ukwiki', 'viwiki')
)

SELECT
  event.editing_session_id as edit_attempt_id,
  event.bucket AS bucket,
  wiki As wiki,
  event.integration AS integration,
  init_events.init_type AS init_type,
  event.anonymous_user_token,
  event.editor_interface AS editor_interface
FROM event.editattemptstep eas
INNER JOIN init_events
ON eas.event.editing_session_id = init_events.edit_attempt_id
AND eas.wiki = init_events.init_wiki
WHERE
  YEAR = 2022
    AND month = 02
    AND day >= 03
-- look at only desktop ready events
  AND event.platform = 'desktop'
  AND event.action = 'ready'
--by anon not bot users
    AND useragent.is_bot = false
    AND event.user_id = 0 
   AND event.user_class = 'IP'
-- only talk page events
  AND event.page_ns % 2 = 1
-- bucketed events only
AND event.bucket in ('test', 'control')
-- test wikis
AND wiki IN ('amwiki', 'bnwiki', 'zhwiki', 'nlwiki', 'arzwiki', 'frwiki', 'hewiki', 'hiwiki',
    'idwiki', 'itwiki', 'jawiki', 'kowiki', 'omwiki', 'fawiki', 'plwiki', 'ptwiki', 'eswiki', 'thwiki',
    'ukwiki', 'viwiki')
Shizhao changed the subtype of this task from "Task" to "Spike".Feb 14 2022, 2:44 AM

Resolving this task as the init_type results were shared and the patch to fix the underlying issue has been created in https://gerrit.wikimedia.org/r/c/mediawiki/extensions/VisualEditor/+/762056/