editattemptstep and web_ui_scroll schema returns null
Closed, ResolvedPublic2 Estimated Story Points
Actions

Assigned To

Authored By

	KSarabia-WMF
	May 22 2023, 8:57 PM

Description

Per conversation with analyst:

For the editattemptstep and mediawiki_web_ui_scroll schema , there are fields of is_bot, skin, wiki in schema. But they are all NULL.

Success Criteria:
Figure out what is causing skin field to return NULL in database, even though the fields are firing with correct values in debug mode and beta.

Query to use:
SELECT *
FROM editattemptstep
WHERE year=2023 and month=5 and day=1
AND wiki='enwiki'
limit 5

Screenshot 2023-06-26 at 8.44.43 AM.png (962×1 px, 307 KB)

QA

Using https://stream-beta.wmflabs.org/v2/ui/#/?streams=mediawiki.web_ab_test_enrollment we manually verified that A/B test enrollment is working.
Using https://stream-beta.wmflabs.org/v2/ui/#/?streams=mediawiki.web_ui_scroll verify that events are coming in and have the schema version 2.0.0. To trigger an event scroll down the page and up again.
Using https://stream-beta.wmflabs.org/v2/ui/#/?streams=eventlogging_EditAttemptStep verify that events are coming in and have the schema version 2.0.0. To trigger an event edit a page on beta cluster.

Details

Other Assignee: Jdlrobson

Subject	Repo	Branch	Lines +/-
Make EventLogging code more robust	mediawiki/extensions/WikiEditor	master	+5 -1
Update reference in WikiEditor	mediawiki/extensions/WikiEditor	master	+7 -1
Editattemptstep event logging	mediawiki/extensions/WikimediaEvents	master	+1 -1
Web ui scroll version update	mediawiki/extensions/WikimediaEvents	master	+1 -1
Update web ui scroll	schemas/event/secondary	master	+187 -4
Updates web enrollment ab test js	mediawiki/extensions/WikimediaEvents	master	+1 -1
Update web_ab_test_enrollment schema	schemas/event/secondary	master	+159 -4
Fix editattemptstep ref	schemas/event/secondary	master	+1 K -329

Customize query in gerrit

Related Objects

Mentioned In: T302358: [A/B Test] Run an A/B test to evaluate impact of Usability Improvements
T335159: Make it possible to figure out what skin someone is using in the A/B test for Talk pages project/Usability

Event Timeline

KSarabia-WMF created this task.May 22 2023, 8:57 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 22 2023, 8:57 PM

KSarabia-WMF added a project: Web-Team-Backlog.May 22 2023, 8:58 PM

Aklapper added a project: Desktop Improvements (Vector 2022).May 23 2023, 1:04 PM

LGoto triaged this task as Low priority.May 25 2023, 5:14 PM

LGoto moved this task from Incoming to Current Fiscal Year on the Web-Team-Backlog board.

MNeisler mentioned this in T335159: Make it possible to figure out what skin someone is using in the A/B test for Talk pages project/Usability.Jun 1 2023, 4:12 PM

MNeisler mentioned this in T302358: [A/B Test] Run an A/B test to evaluate impact of Usability Improvements.Jun 1 2023, 7:23 PM

ppelberg subscribed.Jun 1 2023, 7:25 PM

KSarabia-WMF updated the task description. (Show Details)Jun 5 2023, 7:19 PM

Jdlrobson edited projects, added MediaWiki-extensions-WikimediaEvents; removed Desktop Improvements (Vector 2022).Jun 9 2023, 12:08 AM

KSarabia-WMF updated the task description. (Show Details)Jun 26 2023, 1:45 PM

This is being actively discussed in https://wikimedia.slack.com/archives/CSV483812/p1687563141209929 with @DLynch & @Ottomata and we've identified the root cause. Now figuring out next steps.

@KSarabia-WMF: could you please summarize the learnings from that thread in here?

A quick summary, focused only on EditAttemptStep:

The patch that added these new fields to EditAttemptStep did so by adding the web/common schema fragment to it.
Because EditAttemptStep is a legacy schema (eventcapsule) the fragment-fields got added to the top-level of the schema properties, rather than to the event key's properties. (See: the materialized 1.4.1 schema.)
The legacy schema logging code doesn't expose a way to log data _outside_ the event key, the web/common fragment includes a bunch of other nested fragments that really do need to be at the top-level, _and_ the web/common fragment is included in two non-legacy schemas that won't contain an event key.
Therefore we need to either:
1. Remove the actually-common properties from web/common and put them into a shared fragment (web/common_core?) that can be ref'd into web/common _and_ into the event.properties (via switching it to allOf) of EditAttemptStep
2. Remove the ref of analytics/common from web/common and put it directly into the two non-legacy schema it's used in, so that web/common can be ref'd directly into the event.properties of EditAttemptStep (via converting to allOf again).

I think that the latter option is probably cleaner, since it removes usage of nested fragments that are (as we see here) harder to reason about.

None of this applies to web_ui_scroll, so more investigation may be needed to work out why fields are null there.

+1 for Option B.

KSarabia-WMF moved this task from Current Fiscal Year to Web Team FY2023-24 Q1 Sprint 2 on the Web-Team-Backlog board.Jun 26 2023, 8:51 PM

KSarabia-WMF edited projects, added Web-Team-Backlog (Web Team FY2023-24 Q1 Sprint 2); removed Web-Team-Backlog.

Jdlrobson merged a task: T338178: [SPIKE] Investigate null values in schema.Jun 26 2023, 9:24 PM

Jdlrobson added subscribers: Jdlrobson, ovasileva, Krinkle.

KSarabia-WMF moved this task from Incoming to Doing on the Web-Team-Backlog (Web Team FY2023-24 Q1 Sprint 2) board.Jun 27 2023, 3:20 PM

KSarabia-WMF moved this task from Doing to Incoming on the Web-Team-Backlog (Web Team FY2023-24 Q1 Sprint 2) board.

KSarabia-WMF moved this task from Incoming to Doing on the Web-Team-Backlog (Web Team FY2023-24 Q1 Sprint 2) board.Jun 27 2023, 3:23 PM

Change 934032 had a related patch set uploaded (by Kimberly Sarabia; author: Kimberly Sarabia):

[schemas/event/secondary@master] Fix editattemptstep ref

https://gerrit.wikimedia.org/r/934032

gerritbot added a project: Patch-For-Review.Jun 29 2023, 2:02 AM

Jdlrobson moved this task from Doing to Incoming on the Web-Team-Backlog (Web Team FY2023-24 Q1 Sprint 2) board.Jul 6 2023, 4:55 PM

ovasileva set the point value for this task to 3.Jul 11 2023, 4:51 PM

Change 938280 had a related patch set uploaded (by Kimberly Sarabia; author: Kimberly Sarabia):

[schemas/event/secondary@master] Update web_ab_test_enrollment schema

https://gerrit.wikimedia.org/r/938280

Change 938284 had a related patch set uploaded (by Kimberly Sarabia; author: Kimberly Sarabia):

[schemas/event/secondary@master] Update web ui scroll

https://gerrit.wikimedia.org/r/938284

ovasileva raised the priority of this task from Low to Medium.Jul 17 2023, 5:39 PM

ovasileva changed the point value for this task from 3 to 2.Jul 17 2023, 5:41 PM

ovasileva moved this task from Incoming to Ready for Development on the Web-Team-Backlog (Web Team FY2023-24 Q1 Sprint 2) board.

Change 934032 merged by jenkins-bot:

[schemas/event/secondary@master] Fix editattemptstep ref

https://gerrit.wikimedia.org/r/934032

KSarabia-WMF moved this task from Ready for Development to Code Review on the Web-Team-Backlog (Web Team FY2023-24 Q1 Sprint 2) board.Jul 18 2023, 4:36 PM

Jdlrobson updated Other Assignee, added: Jdlrobson.Jul 18 2023, 4:42 PM

Change 938280 merged by jenkins-bot:

[schemas/event/secondary@master] Update web_ab_test_enrollment schema

https://gerrit.wikimedia.org/r/938280

Change 939389 had a related patch set uploaded (by Kimberly Sarabia; author: Kimberly Sarabia):

[mediawiki/extensions/WikimediaEvents@master] Updates web enrollment ab test js

https://gerrit.wikimedia.org/r/939389

Change 939389 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] Updates web enrollment ab test js

https://gerrit.wikimedia.org/r/939389

I believe remaining work is

Equivalent of https://gerrit.wikimedia.org/r/939389 for the WebScroll schema
Equivalent of https://gerrit.wikimedia.org/r/939389 for the editattemptstep schemas.

ReleaseTaggerBot added a project: MW-1.41-notes (1.41.0-wmf.19; 2023-07-25).Jul 19 2023, 1:00 AM

Change 939762 had a related patch set uploaded (by Kimberly Sarabia; author: Kimberly Sarabia):

[mediawiki/extensions/WikimediaEvents@master] Web ui scroll version update

https://gerrit.wikimedia.org/r/939762

Change 938284 merged by jenkins-bot:

[schemas/event/secondary@master] Update web ui scroll

https://gerrit.wikimedia.org/r/938284

Jdlrobson reassigned this task from KSarabia-WMF to Edtadros.Jul 19 2023, 8:56 PM

Jdlrobson moved this task from Needs More Work to QA on the Web-Team-Backlog (Web Team FY2023-24 Q1 Sprint 2) board.

Jdlrobson updated the task description. (Show Details)

Change 939762 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] Web ui scroll version update

https://gerrit.wikimedia.org/r/939762

Maintenance_bot removed a project: Patch-For-Review.Jul 20 2023, 6:30 PM

Change 940239 had a related patch set uploaded (by Kimberly Sarabia; author: Kimberly Sarabia):

[mediawiki/extensions/WikimediaEvents@master] Editattemptstep event logging

https://gerrit.wikimedia.org/r/940239

gerritbot added a project: Patch-For-Review.Jul 20 2023, 6:40 PM

Jdlrobson updated the task description. (Show Details)Jul 20 2023, 7:00 PM

Change 940239 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] Editattemptstep event logging

https://gerrit.wikimedia.org/r/940239

The reference in WikiEditor will also need to be updated.

Maintenance_bot removed a project: Patch-For-Review.Jul 20 2023, 7:10 PM

...and possibly updated to actually log this data if you want it from there, since it looks like it never got updated to log the web_common fields.

@DLynch Gotcha. Will do. Thanks again for looking out.

Change 940255 had a related patch set uploaded (by Kimberly Sarabia; author: Kimberly Sarabia):

[mediawiki/extensions/WikiEditor@master] Update reference in WikiEditor

https://gerrit.wikimedia.org/r/940255

gerritbot added a project: Patch-For-Review.Jul 20 2023, 9:22 PM

Test Result - Beta

Status: ✅ PASS
Environment: beta
OS: macOS Ventura
Browser: Chrome
Device: MBP
Emulated Device:NA

Test Artifact(s):

QA Steps

✅ AC1: Using https://stream-beta.wmflabs.org/v2/ui/#/?streams=mediawiki.web_ab_test_enrollment we manually verified that A/B test enrollment is working.
This was manually verified by the engineers

✅ AC2: Using https://stream-beta.wmflabs.org/v2/ui/#/?streams=mediawiki.web_ui_scroll verify that events are coming in and have the schema version 2.0.0. To trigger an event scroll down the page and up again.

wiki: wikifunctionswiki
skin: vector-2022
is_bot: false
$schema: /analytics/mediawiki/web_ui_scroll/2.0.0
web_session_id: 81573b86cab2bca41d4a
page_id: 2247
is_anon: false
action: scroll-to-top
access_method: desktop
dt: '2023-07-20T22:33:55.896Z'
meta:
  stream: mediawiki.web_ui_scroll
  domain: wikifunctions.beta.wmflabs.org
  id: 04c2a578-f504-4207-8ace-1407a11c9f55
  dt: '2023-07-20T22:34:29.710Z'
  request_id: 9728e9b0-274d-11ee-9931-897bd4c4e7d2
  topic: eqiad.mediawiki.web_ui_scroll
  partition: 0
  offset: 18694
http:
  request_headers:
    user-agent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'

✅ AC3: Using https://stream-beta.wmflabs.org/v2/ui/#/?streams=eventlogging_EditAttemptStep verify that events are coming in and have the schema version 2.0.0. To trigger an event edit a page on beta cluster.

Logged In

event:
  page_id: 89497
  page_title: Hooded_skunk
  page_ns: 0
  revision_id: 595222
  user_id: 16155
  user_is_temp: false
  user_editcount: 88
  mw_version: 1.41.0-alpha
  page_token: 42575a9c5c02e37fb03e
  session_token: d448f95d6f4d5bd446d5
  version: 1
  wiki: enwiki
  skin: vector-2022
  is_bot: false
  editing_session_id: 0aca34a7376d56fe056a
  editor_interface: visualeditor
  integration: page
  platform: desktop
  action: init
  init_type: page
  init_mechanism: click
  is_oversample: false
schema: EditAttemptStep
webHost: en.wikipedia.beta.wmflabs.org
wiki: enwiki
$schema: /analytics/legacy/editattemptstep/2.0.0
client_dt: '2023-07-21T00:50:01.026Z'
meta:
  stream: eventlogging_EditAttemptStep
  domain: en.wikipedia.beta.wmflabs.org
  id: e61c5d1c-23bf-463a-897f-f4d573d60070
  dt: '2023-07-21T00:50:31.277Z'
  request_id: 97d4bf70-2760-11ee-9931-897bd4c4e7d2
  topic: eventlogging_EditAttemptStep
  partition: 0
  offset: 447307
dt: '2023-07-21T00:50:31.277Z'
http:
  request_headers:
    user-agent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'
  client_ip: 47.157.254.85

Anon

event:
  page_id: 89497
  page_title: Hooded_skunk
  page_ns: 0
  revision_id: 595222
  user_id: 0
  user_is_temp: false
  user_class: IP
  user_editcount: 0
  mw_version: 1.41.0-alpha
  page_token: 7417ce194fc7f9d87956
  session_token: d448f95d6f4d5bd446d5
  version: 1
  wiki: enwiki
  skin: vector-2022
  is_bot: false
  editing_session_id: 68e8afb5f26b6f588c10
  editor_interface: visualeditor
  integration: page
  platform: desktop
  action: init
  init_type: page
  init_mechanism: click
  is_oversample: false
schema: EditAttemptStep
webHost: en.wikipedia.beta.wmflabs.org
wiki: enwiki
$schema: /analytics/legacy/editattemptstep/2.0.0
client_dt: '2023-07-21T00:51:38.732Z'
meta:
  stream: eventlogging_EditAttemptStep
  domain: en.wikipedia.beta.wmflabs.org
  id: 3417bbad-26b1-4f44-a414-76b922b2e350
  dt: '2023-07-21T00:51:49.759Z'
  request_id: c69c9bc0-2760-11ee-9931-897bd4c4e7d2
  topic: eventlogging_EditAttemptStep
  partition: 0
  offset: 447316
dt: '2023-07-21T00:51:49.759Z'
http:
  request_headers:
    user-agent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'
  client_ip: 47.157.254.85

KSarabia-WMF assigned this task to jwang.Jul 25 2023, 4:50 PM

Change 940255 merged by jenkins-bot:

[mediawiki/extensions/WikiEditor@master] Update reference in WikiEditor

https://gerrit.wikimedia.org/r/940255

Krinkle unsubscribed.Jul 26 2023, 5:04 PM

Maintenance_bot removed a project: Patch-For-Review.Jul 26 2023, 5:10 PM

ReleaseTaggerBot edited projects, added MW-1.41-notes (1.41.0-wmf.20; 2023-08-01); removed MW-1.41-notes (1.41.0-wmf.19; 2023-07-25).Jul 26 2023, 6:00 PM

jwang added a project: Product-Analytics.Jul 27 2023, 5:38 PM

Change 942677 had a related patch set uploaded (by Thiemo Kreuz (WMDE); author: Thiemo Kreuz (WMDE)):

[mediawiki/extensions/WikiEditor@master] Make EventLogging code more robust

https://gerrit.wikimedia.org/r/942677

gerritbot added a project: Patch-For-Review.Jul 28 2023, 2:49 PM

@jwang

Please sign off on this when you have a moment by querying the following and confirming is_bot and is_anon is no longer NULL

select * from mediawiki_web_ui_scroll
where year = 2023
and month = 7 and day = 27
and wiki = 'enwiki' order by hour desc limit 5

select * from mediawiki_web_ab_test_enrollment
where year = 2023
and month = 7 and day = 27
and wiki = 'enwiki' order by hour desc limit 5

*FYI - For anyone wondering, the AB test is off and I confirmed it again today by creating a new account and seeing the enrollment event is no longer firing, but there are some heavy caching some places that's causing some enrollment events to still fire.

select * from editattemptstep
where year = 2023
and month = 7 and day = 27
and wiki = 'enwiki' order by hour desc limit 5

*FYI - This is still returning null in the database. You and I talked about this and the edit schemas seem to have a lot of null values in the db no matter what we do, but we can still check is_bot and is_anon in the event object.

Have verified editattemptstep schema that skin and is_bot are captured in event field. But is_anon is NULL.

query

SELECT event.skin, event.is_bot
FROM editattemptstep
WHERE year=2023 and month=7 and day=28
AND wiki='enwiki'
LIMIT 5

Have verified mediawiki_web_ui_scroll schema that is_bot and skin are captured with value.

query

SELECT is_bot, skin,wiki, is_anon
FROM mediawiki_web_ui_scroll 
WHERE year=2023 and month=7 and day=28
AND wiki='enwiki'
LIMIT 5

thanks everyone!

For reference, is_anon was added in the recently-ref'd web_common fragment, and in EditAttemptStep is redundant with the user_class field (which can be null or IP). As such, although adding is_anon would fill things out, it's not actually required for anything.

Change 942677 merged by jenkins-bot: